WH ITE PAPER
Index
• What is Fixed-Mobile Convergence? • Consumer Satisfaction Drives FMC Market Growth • Real-world Wi-Fi Mobility is Critical • User Experience Identifies Critical Mobile Performance Scenarios • Wi-Fi Mobility Performance Testing • Effective Test Methodologies • Overview of Azimuth FMC Performance Test Solution • Summary
Consumer Satisfaction with FixedMobile Convergence Starts with Mobility Performance Testing
The rich availability of Wi-Fi connectivity, along with universal demand for constant access to voice and data communication is driving consumer and enterprise demand for converged Wi-Fi/Cellular applications and services. Under the banner of Fixed-Mobile Convergence (FMC), these applications and services promise better access to voice and data services as well as lower communication costs. Successful delivery of these converged services is based on the assumption that wireless IP networks and 802.11a/b/g/n devices can deliver the underlying service necessary to guarantee a satisfying end-user experience. This whitepaper begins with an overview of the FMC market and end-user experience that is required to drive the market for FMC services. The whitepaper will then discuss the Wi-Fi mobility usage scenarios that are critical to FMC handset operation and how real-world environmental conditions can impact user experience. Finally, the whitepaper will propose efficient methods for testing Wi-Fi mobility performance.
What is Fixed-Mobile Convergence?
The term Fixed-Mobile Convergence (FMC) means different things to different groups. For end-users, it is the promise of quality fixed services and applications (voice, video and data) being delivered seamlessly over mobile (wireless) networks to handsets or endpoint devices. For infrastructure vendors, FMC represents a large commercial opportunity to deliver back-end integration products and services and for service providers, it represents an opportunity to develop additional revenue from existing cellular and/or Wi-Fi network infrastructure investments. To drive increased adoption by end-users, who will ultimately dictate the success or failure of the FMC market, the industry must deliver compelling FMC applications and services. The primary FMC usage scenario expected by end-users is the seamless transference of voice calls across cellular and Wi-Fi networks. In this scenario, users will be able to make a call on their cellular network while outside an office building/home environment and have the call seamlessly transferred to a Wi-Fi network when they move inside their office building/home. With the voice call seamlessly transferred from the cellular network to the corporate/home network, users enjoy continuous communication and benefit from improved coverage of the Wi-Fi network, while also reducing cellular minute usage as a result of the call being transferred off the cellular network. Alternatively, moving voice calls from a corporate or home network to a cellular network enables continuous communication when users move outside their office building or home.
www.azimuthsystems.com
Location independent access to data applications also represents a huge draw for potential users of FMC services. While stationary or mobile in any location (home, office, airport, conference center, car, train, airplane, etc.) that is covered by a cellular or Wi-Fi network, users will have access to internet-based applications such as email, web browsing and online services like banking and investing, as well as news and entertainment. This ability to utilize data applications where and when they want will enable connected users to benefit from significant improvements in personal and professional productivity.
Consumer Satisfaction Drives FMC Market Growth
Sustainable consumer demand for FMC services will be driven by three major factors: the ability to deliver continuous voice and data communication; the ability to deliver cost savings that are not available from alternative Wi-Fi and cellular-only services; and the ability to deliver a satisfying end-user experience. Existing cellular services have established user experience expectations that new converged services must meet or exceed. For service providers, this means FMC must deliver “carriergrade” voice and data services characterized by:
• • • • •
Good voice quality; Few dropped calls; Reliable and high throughput data connectivity; Long handset battery life in standby and active states; Seamless voice and data roaming/handoff.
A failure to deliver a satisfying end-user experience will provide consumers with little motivation to adopt converged Wi-Fi/Cellular services. Instead, users will continue to use their cellular and/or Wi-Fi -only services. The end result will be the inability of the industry to generate sustainable demand for FMC services.
Real-world Wi-Fi Mobility is Critical
The growth in consumer demand for services delivered by long range cellular networks has been driven primarily by improvements in mobile voice services and value-added data services like text messaging and email/web access. FMC broadens the range of this capability into areas where cellular coverage may be less reliable (indoors) and onto the Wi-Fi network, which has the potential for much higher speed data services. To foster demand for converged Wi-Fi/Cellular FMC services, the addition of comparatively short range Wi-Fi connectivity must at least sustain the mobility and quality of service that cellular users have come to expect. Converged Wi-Fi/Cellular services must support the same or better quality voice and data services while the user is in motion and transferring from Wi-Fi to cellular networks (and vice versa). This mobility service requires adequate performance in three key usage scenarios: 1. Wi-Fi Range – Quality voice and data services while the user moves closer to and away from a Wi-Fi Access Point (AP); 2. Wi-Fi Roaming – Quality voice and data services while the user is moving between Wi-Fi APs; 3. Hand-in/Hand-out network handover– Quality voice and data services while moving from Wi-Fi to cellular networks or vice versa;
Page 2 – Consumer Satisfaction with Fixed-Mobile Convergence Starts with Mobility Performance Testing
Wi-Fi range, or coverage, is a significant measure of Wi-Fi performance. Due to lower transmit power on the Wi-Fi AP and higher frequency, Wi-Fi has a much shorter range than cellular technology. In contrast to cellular technology, Wi-Fi technology varies the transmission data rate to minimize packet errors. A Wi-Fi transmitter, be it on the AP or client, uses dynamic rate adaptation algorithms to control the transmission data rate on a packet by packet basis. These algorithms consider many different network and environmental variables including receive signal strength and packet error rate in deciding to increase or decrease the transmission rate. If these algorithms are poorly implemented, movement of the user around a Wi-Fi AP will significantly impact the data throughput and voice quality that the user experiences. Proving that the rate adaptation algorithms have been implemented properly requires extensive range testing of the client device. Wi-Fi roaming is another critical measure of Wi-Fi device performance. Both cellular and Wi-Fi networks use algorithms to transfer connectivity from one infrastructure device (cellular base station, Wi-Fi AP, etc.) as connection conditions require. Cellular networks utilize roaming algorithms that are implemented on the base station and managed by the network, which make roaming decisions in consideration of real-time network conditions (base station loading, base station service outage, etc). Wi-Fi networks, in contrast, utilize roaming algorithms implemented on the client, which makes roam decisions without consideration of real-time network conditions (i.e. Is the AP that I want to roam to overloaded with too many clients? Would it be better to roam to an alternate AP that is not heavily loaded”?). This means that Wi-Fi clients could make decisions to roam to an AP that is overloaded or not functioning properly, which would result in significantly reduced data throughput and voice quality or the session being terminated completely. In addition to the differences in the implementation of the roaming algorithm, cellular and Wi-Fi networks execute the roam differently. Cellular networks execute the roam using a “make before break” approach which establishes the connection between the handset and base station that is being given control of the session prior to terminating the connection between the handset and the base station that is giving up control of the session. In contrast, Wi-Fi clients utilize a “break before make” approach to executing the roam, in which the client terminates connectivity with one AP prior to establishing connectivity with a new AP. In the event that the client takes a long time to establish the connection with the new AP or fails to make the connection, data throughput, voice quality and call continuity can be impacted significantly. Given the differences in the roaming algorithm implementation and execution, proving that Wi-Fi roaming is conducted efficiently to maintain data throughput, voice quality and satisfactory call continuity (low number of dropped calls) requires extensive testing. Transferring session connectivity between Wi-Fi and cellular networks (hand-in/hand-out handover) is much more complex than a Wi-Fi to Wi-Fi roam or a cellular to cellular handover. With the differences between cellular (base station-side, network managed) and Wi-Fi (client-side) handover/roaming algorithm implementations, engineering efficient Wi-Fi to cellular handovers requires complex changes to the back-end cellular network, as well as significant changes to decision making algorithms. With this increased complexity, it is also critical that extensive testing be done to validate the impact of handover on throughput, voice quality and call continuity.
Page 3 – Consumer Satisfaction with Fixed-Mobile Convergence Starts with Mobility Performance Testing
When discussing the performance of Wi-Fi devices, the impact of real-world environment conditions in which the devices operate must be recognized. Unlike a cellular network, which is the sole occupier of the licensed RF spectrum in which it operates, a Wi-Fi network operates in unlicensed spectrum that it shares not only with other Wi-Fi networks, but potentially with other RF networks like Bluetooth- or even other RF devices such as cordless phones and microwave ovens. The presence of RF devices and Wi-Fi networks competing for spectrum causes RF interference that can significantly impact the performance of a Wi-Fi network. In addition, solid obstacles such as walls and furniture, as well as the movement of objects such as vehicles, can create RF signal conditions, known as multipath and fading, that impact the performance of Wi-Fi devices. Finally, most FMC solutions will utilize the Internet, a non-dedicated IP network, as a primary carrier of the voice data, which can directly affect voice quality as traffic load varies. To maintain the best possible FMC services, converged Wi-Fi/Cellular handsets must deliver the best Wi-Fi mobility performance in all these different types of real-world conditions.
User Experience Identifies Critical Mobile Performance Scenarios
As discussed earlier, the ability of converged Wi-Fi/Cellular devices to deliver the realworld range, roaming and hand-in/hand-out performance required to support “carriergrade” FMC services must be ensured through extensive testing. To determine the impact of Wi-Fi performance on user experience, we need to first identify the critical mobile performance scenarios that directly impact the user experience. Table 1 below identifies critical mobile performance scenarios.
Table 1: Critical Mobile Performance Scenarios User Experience
Longer time required to download large file when in cafeteria
Mobile Performance Scenarios
Data throughput over range
Clear voice call with customer while walking from office to conference room Connectivity lost during data download when moving between offices Call breaks up or has poor voice quality when going over to the other part of the building to ask Mary about the shipment Call continuity and voice quality when walking from office building to car Call clarity is bad in manufacturing area Pick up phone after meeting to call home and the battery is dead
Voice quality over range
Data throughput during roaming
Dropped calls during roaming Voice quality during roaming Dropped calls during hand-in/hand-out network transfer Voice quality during hand-in/hand-out network transfer Interference from microwave ovens Battery life caused by customer’s Wi-Fi network activity throughout the meeting
Page 4 – Consumer Satisfaction with Fixed-Mobile Convergence Starts with Mobility Performance Testing
Wi-Fi Mobility Performance Testing
The mobile performance scenarios identified earlier establish a framework for the Wi-Fi mobility testing that is required. These mobile performance scenarios establish the fundamental Wi-Fi performance metrics including data rate, packet loss, error rate, and roam time that must be tested. A set of essential tests for the evaluation of the Wi-Fi mobility performance can be developed as listed in Table 2 below.
Table 2: Required Wi-Fi Mobility Performance Tests Mobile Performance Scenario
Data throughput over range
Wi-Fi Performance Metric
Data Rate Packet Loss Error Rate Packet Loss
Wi-Fi Performance Test
Throughput (Megabits per second – Mbps) as a function of distance from a Wi-Fi AP (Example: position of your office relative to AP)
Voice quality over range
Voice quality measurements - Perceptual Evaluation of Speech Quality (PESQ)- based MOS scores - as a function of distance from the Wi-Fi AP (Example: talking on phone in front of TV or in kitchen relative to AP) Roam time in seconds (s) for fail-over roam as handset is forced to move to a new AP because one has failed (Example: power glitch or network failure) Roam time (s) for smooth roam as handset moves between two different Wi-Fi APs (Example: walking between work areas)
Data AP roaming
Roam Time Data Rate
VoWi-Fi handover
Roam Time Packet Loss Error Rate
Number and percentage of voice calls dropped as handset moves between two different Wi-Fi APs or transfers from Wi-Fi to cellular networks (and vice versa) (Example: walking down to another part of the building while talking on the phone or moving outside the building Voice quality measurements - PESQ-based MOS scores, when handset moves between two different APs or transfers from Wi-Fi to cellular networks (and vice versa) (Example: maintain call quality when moving around a house or when moving from garden into house)
Effective Test Methodologies
The market for FMC applications and services encompasses a thriving ecosystem of handsets, PDAs, portable media players and other consumer electronics devices that must deliver the robust mobility performance necessary to provide a quality end-user experience. Ensuring that these devices and the networks on which they operate are capable of delivering the necessary performance requires extensive testing. For mobility testing to be effective from a quality, coverage and cost perspective, test methodologies need to accurately recreate the “real world” in an efficient, repeatable, cost effective and scalable manner. One methodology to test Wi-Fi mobility performance is Over-The-Air (OTA) testing. In OTA testing, engineers test device performance by replicating the actual conditions of the environments in which they will operate. This is accomplished by renting or buying empty office
Page 5 – Consumer Satisfaction with Fixed-Mobile Convergence Starts with Mobility Performance Testing
buildings, homes, or even testing on live networks. Mobility and roaming are tested by placing Wi-Fi devices on mobile carts, moving these carts to various locations in the test space, and manually configuring tests and recording test results at each location. Due to the uncontrollable nature of OTA environmental conditions, the majority of testing is done manually. The effectiveness of Wi-Fi mobility performance testing using OTA methods is limited by two critical factors. First, the time consuming manual test setup and execution typical of OTA tests limits the ability of this testing method to scale. Second, consistent, repeatable test measurements are nearly impossible in open-air environments, limiting the ability to reliably repeat the tests in the future. In addition, the RF interference may vary with each test iteration at the same location, which may make reproducing results and issues nearly impossible. The alternative to OTA testing is testing in a controlled RF environment. One such example of a controlled RF testing can be done using either environment/room isolation. Room isolation, which sees the WLAN test setup placed in a screen room that filters out external RF interference, is costly due to the large installation and maintenance costs of the screen rooms. In addition, the size of the screen room severely limits the effectiveness of testing distance, roaming and mobility performance. A more advanced method of controlled RF testing involves device isolation. In device isolation, all test bed devices are placed in individual isolated enclosures and are connected via cables to programmable RF attenuators, combiners, and switches. This test methodology replicates the Wi-Fi network in a controlled, cabled environment that stabilizes the RF connection by removing the variability of open-air systems. This approach provides a completely controllable test bed. There are numerous benefits to a Wi-Fi testing methodology that utilizes device isolation. Isolation of Devices Under Test (DUTs) from external RF interference provides a controllable RF environment to conduct repeatable mobility testing. Test solutions that utilize a controlled, cabled RF environment reduce test costs by eliminating the need to design, build, and maintain home-grown test beds and costly RF screen rooms. Another benefit of this test methodology comes from the programmable test bed and tools which enable automated test configuration and execution. To analyze the effect of mobility on both device and network performance, users can automatically configure any network device and dynamically position any network node. Automated test configuration supports the simple, effective setup and re-use of test configurations, allowing for repeatable test execution over time. Using programming, scripts can be created which require little human intervention and can automatically run multiple iterations of different configurations in a fraction of the time that is required for manual testing. This repeatability reduces the time spent on quality assurance and benchmark test processes and as a result, dramatically reduces time to market and testing costs. Test scalability is another important benefit of this approach. If the controlled RF environment is properly architected, system designers can scale Wi-Fi testing from a single device to the entire network. Users can configure an entire Wi-Fi network and provide systemlevel testing of actual APs, clients, and other wireless devices. Networks can be tested
Page 6 – Consumer Satisfaction with Fixed-Mobile Convergence Starts with Mobility Performance Testing
under a variety of traffic and client load conditions. Client and traffic load emulation enables the development of test setups that re-create a busy network environment for the DUTs. An additional benefit of this approach is the ability to test Wi-Fi mobility performance in controlled real-world network conditions. Injecting impairments into an isolated RF environment enables the emulation of controlled real-world wireless networking conditions. Using this test approach, engineers can assess the impact of one or a combination of the following real-world conditions: RF multipath and fading; background Wi-Fi traffic; RF interference; and IP network delay, on the mobility performance of 802.11a/b/g/n devices. FMC developers, users and service providers have a variety of testing requirements which can be addressed with this test methodology. Starting from a development environment where controlled isolation in a crowded RF lab can be invaluable, to integration where validation of more advanced functions like roaming and security are critical and QA where hundreds of test scenarios and iterations need to be run on every model and version. Another critical test process is conducted by systems engineering groups within service provider organizations, who will use these same test scenarios to validate interoperability of devices from different suppliers, benchmark performance of different configurations to make purchasing decisions and certify devices for deployment. For engineers tasked with selecting FMC handsets that will operate on service provider networks, an effective performance benchmarking process provides a means of performing an “apples to apples” comparison of the Wi-Fi mobility performance delivered by FMC handsets from different manufacturers. To summarize, the most effective method for testing the mobility performance of 802.11a/b/g/n devices utilizes a controlled, cabled RF environment to provide accurate and repeatable test results, leverages test automation to reduce test time as well as costs, and utilizes programmable test tools and external test components such as dynamic channel emulators, Wi-Fi traffic and RF signal generators and network simulators to analyze mobility performance in controlled, real-world network conditions.
Overview of Azimuth FMC Performance Test Solution
The Azimuth FMC Performance Test Solution provides a set of comprehensive automated scripts that streamline the testing of FMC handsets and converged wireless devices by carriers, handset manufacturers and semiconductor vendors. The FMC Performance Test Suite uses sophisticated motion emulation and automated traffic generation control in an isolated environment to ensure accurate and repeatable data and voice Wi-Fi mobility performance testing of Wi-Fi/Cellular handsets over a broad range of user environments. Enhancements to the Azimuth FMC Performance Test Solution enable the insertion of varying RF impairments (background Wi-Fi traffic, RF interference, multipath channel, fading) and IP Network impairment conditions (randomized packet loss, delay and jitter). For more information on the Azimuth FMC Performance Test Solution, please visit http://www.azimuthsystems.com/fmc-performance-test-suite.htm
Page 7 – Consumer Satisfaction with Fixed-Mobile Convergence Starts with Mobility Performance Testing
The Azimuth FMC Performance Test Solution’s family of scripts is presented in Table 3 below.
Table 3: Descriptions of Azimuth FMC Performance Test Scripts Test Script
VoWi-Fi Handset Range Client Range Performance
Description
Assesses handset voice quality using the PESQ MOS scores as a function of distance from the AP Analyzes the throughput and automatic rate adaptation performance of a device as a function of its distance from the AP At each fixed data rate, this script measures the throughput of the device as a function of range from the AP to produce a rate adaptation benchmark graph Measures call integrity (dropped calls) and voice quality by creating an accurate emulation of a roaming setting through precise control of the motion between the handset and two different Wi-Fi APs. Measures roam time for fail-over or smooth transition roam by creating an accurate emulation of a roaming setting through precise control of the motion between the handset and two different Wi-Fi APs. Measures number of calls dropped and voice quality during handover between Wi-Fi and cellular networks in varying network conditions. Places the device under test into a set mode or series of mode sequences and measures the time for the battery to run to exhaustion. The key metrics of this test are standby time and talk time.
Rate Sensitivity Performance
VoWi-Fi Handset Roaming
Client Roaming Performance
VoWi-Fi Handset Hand-in/ Hand-out Performance
Battery Life Performance
Figure 1 below is a detailed schematic of the typical FMC test configuration and shows the components of the Azimuth FMC Performance Test Solution.
Fig. 1: Components of Azimuth FMC Performance Test Solution
Page 8 – Consumer Satisfaction with Fixed-Mobile Convergence Starts with Mobility Performance Testing
Summary
Converged Wi-Fi/Cellular FMC applications and services promise benefits ranging from continuous access to voice and data applications and services to reduced communication costs. Sustainable consumer demand will be developed for “carrier-grade” FMC services that deliver reliable and fast data throughput, good voice quality, few dropped calls and long handset battery life. With cellular-only services proven to deliver voice and data services with “carrier-grade” quality, the ability to deliver FMC services of similar quality is heavily dependent upon on the performance delivered by Wi-Fi-enabled devices as users move within and between Wi-Fi networks, as well as transfer between Wi-Fi and cellular networks. Critical FMC performance metrics that directly affect user experience are measured by voice quality, dropped calls and battery life. These user experience metrics are directly impacted by Wi-Fi data rate, error rate, packet loss and roam time metrics that define Wi-Fi range, roaming and hand-in/hand-out mobility performance. Azimuth Systems delivers the most effective and comprehensive platform for testing the real-world mobility performance of 802.11a/b/g/n devices. Azimuth’s FMC Solution utilizes a controlled, cabled RF environment for repeatable test results, leverages test automation to reduce test time, effort as well as cost and utilizes programmable test tools to enable effective performance benchmark performance in controlled, real-world network conditions.
About Azimuth Systems
Azimuth Systems is a leading provider of wireless data communications test solutions. Azimuth’s products are used by the world’s foremost wireless semiconductor, system vendors and service providers to speed time-to-market and improve wireless product quality.
Corporate Headquarters
Azimuth Systems, Inc. 31 Nagog Park Acton, MA 01720 Phone: 978. 263. 6610 info@azimuthsystems.com
West Coast
Azimuth Systems, Inc. 2890 Zanker Road, Suite 205 San Jose, CA 95134 Phone: 408. 943. 8300 www.azimuthsystems.com
©2007 Azimuth Systems, Inc. All rights reserved. All specifications subject to change without notice. Azimuth, the Azimuth Logo, and ACE are trademarks of Azimuth Systems, Inc. All other trademarks are the property of their respective holders.
Page 9 – Consumer Satisfaction with Fixed-Mobile Convergence Starts with Mobility Performance Testing
Consumer_Satisfaction_with_FMC 0907